Method and device for automatically drawing structural cracks and precisely measuring widths thereof

ABSTRACT

The present invention discloses a method and device for automatically drawing structural cracks and precisely measuring widths thereof. The method comprises a method for automatically drawing cracks and a method for calculating widths of these cracks based on a single-pixel skeleton and Zernike orthogonal moments, wherein the method for automatically drawing cracks is used to rapidly and precisely draw cracks in the surface of a structure, and the method for calculating widths of these cracks based on a single-pixel skeleton and Zernike orthogonal moments is used to calculate widths of macro-cracks and micro-cracks in an image in a real-time manner.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a 371 of international application of PCTapplication serial no. PCT/CN2018/078104, filed on Mar. 6, 2018, whichclaims the priority benefit of China application no. 201810006120.0,filed on Jan. 3, 2018. The entirety of each of the above mentionedpatent applications is hereby incorporated by reference herein and madea part of this specification.

BACKGROUND Technical Field

The present invention relates to the field of structure detection andevaluation, and in particular to a method and device for detectingstructural cracks on a surface based on digital image processing.

Description of Related Art

In the field of civil engineering structure, the test for bearingcapacity of a new material or structure is one of the experimentalmethods that are most widely used. Besides structural response indexessuch as structural strain and deformation, the development of astructural crack, having a clear correspondence to the bearing capacityof a structure, is an important part of a structural bearing capacitytest. However, at present, cracks are mainly observed manually, anddrawn by a person with interrupting an experiment, and widths of thecracks are measured by manually attaching a crack width gauge to thesurface of a structure. In addition, crack condition is one of the mainindexes for periodic structure inspection, load test and laboratoryexperiment. Manually drawing cracks and measuring crack widths with acrack width gauge affects the progress of an experiment due to lowefficiency as well as a waste of time and energy, and also lead to adetection result of great personal error.

In recent years, the concrete crack detection technology based ondigital image processing, which can overcome disadvantages ofconventional manual detection, such as a waste of time and energy aswell as inadequate detection precision, has become a hotspot forresearch of crack detection technologies at home and abroad. Commonimage-based crack identification methods include edge detection,threshold segmentation and so on. For example, a method and system forpavement crack identification based on Prewitt operator disclosed inChinese Patent Publication No. CN 106651872A and a method for pavementcrack image detection based on Hessian matrix multi-scale filteringdisclosed in Chinese Patent Publication No. CN 105719283 are all basedon conventional image processing algorithms, however, as the backgroundand features of a crack have considerable variability and complexity, itis difficult to determine whether it is a crack according to featuresidentified by a conventional algorithm. Moreover, a conventionalalgorithm tends to require more human intervention, thereby resulting ina low automation level.

Since Hinton published an article in Science in 2006, more and moreattention has been paid to deep learning. Deep learning has achievedamazing success in image recognition and has also started to be appliedto crack identification based on image processing. Abroad, Cha,Young-Jin and others have applied deep learning to crack identificationbased on image processing. In China, Chinese Patent Publication No. CN106910186A discloses a method for detecting and locating bridge cracksbased on CNN deep learning, which can hardly meet the requirements fordetecting cracks of different widths in an image due to adopting CNNtemplates of only 16 pixels in size, and cannot output binary images ofcracks accurate to pixel level. Chinese Patent Publication No. CN107133960A discloses an image crack segmentation method based on a deepconvolutional neural network, which utilizes a feature map obtained bythe convolutional network as an input to carry out crack regionsegmentation. As the size of a pixel block is generally much smallerthan the size of the whole image, the size of the pixel block will limitthe size of a sensed region, and only some local features can beextracted. Therefore, this method has disadvantages such as limitedperformance of classifying, poor sensitivity of an identification resultto details in an image, and a detection result of inadequate precisiondue to a fuzzy crack edge. The method can hardly be applied to realprojects.

Moreover, all conventional crack width measurement methods based ondigital image processing calculate crack width as the distance betweentwo pixels, and thus have great limitations. For example, Chinese PatentPublication No. CN 106018411A discloses a method and device formeasuring crack width, wherein crack width is measured by calculatingthe resolution of an image according to the size of a laser spot emittedby a laser in the image and then multiplying the resolution by thedistance between two pixels. Chinese Patent Publication No. CN104089580A discloses an instrument and method for measuring width of acrack on a concrete surface based on a smartphone, wherein crack widthis measured by calculating a pixel resolution by a manual markingmethod, and then multiplying the resolution by an average distanceobtained from “pixel counting” between two edges. Crack widthmeasurement methods based on “pixel counting” in the past are oftenapplicable to macro-cracks in images and can work well. However, formicro-cracks (smaller than 5 pixels) in images, these methods have greatidentification errors and can hardly meet requirements for measurementprecision.

In general, crack condition is an important detection index for periodicstructure inspection, laboratory experiments and other projects. Themethod for manually drawing cracks is time-consuming and laborious, andhas a great error in measurement of crack width. Moreover, there arestill great defects in crack detection based on image processing. Thesedefects are specifically as follows: (1) the conventional automaticcrack drawing technology based on image processing tends to requireexcessive human interference, so that the automation level is low andcracks cannot be effectively distinguished from noise interferences, andalthough the existing image crack identification technology based ondeep learning has achieved some success, it still cannot meetrequirements for automatically and precisely drawing a crack; and (2)the calculated width of a micro-crack in an image is of great error, andit needs to sacrifice the size of the field of view to meet requirementsfor detection precision.

SUMMARY

The present invention discloses a method and device for automaticallydrawing structural cracks and precisely measuring widths thereof tosolve technical problems existing in the prior art. The method aims atrapidly and automatically drawing structural cracks and preciselymeasuring widths thereof. While automatically drawing cracks in an imagewith high precision, high accuracy and high recall, the method canaccurately measure widths of both macro-cracks and micro-cracks smallerthan 5 pixels in the image.

In order to achieve the aforementioned objects, the overall idea of thepresent invention is as follows:

A method for automatically drawing structural cracks comprises thefollowing steps: 1) detecting, by a crack detection method based onmulti-scale deep learning, a crack range in an image using multi-scaledeep learning to ensure that no crack information is missed forsubsequent work; 2) extracting, by a preliminary crack extraction methodof “global-local co-segmentation” based on deep learning, medianfiltering and Hessian matrix-based linear enhancement, full informationof both macro-cracks and micro-cracks in the image to ensure a detectionresult of high recall; and 3) effectively removing, by a fine crackdetection method based on deep learning, image segmentation and imagereconstruction, noise from the result to ensure a detection result ofhigh accuracy and precision.

As an improvement of the present invention, the crack detection methodbased on multi-scale deep learning in step 1) is specifically asfollows:

Because concrete cracks are complex and diversified in morphology,single-scale deep learning often cannot meet requirements for detectingcracks of different widths in an image, and crack information in animage can be easily missed in the analysis based on detection results ofsingle-scale deep learning. With deep learning models of multiple scalesfrom large to small, the large-scale deep learning modules are firstadopted to scan and detect the whole image; and on the basis that acrack range has been determined by the large-scale (224*224*3) deeplearning modules, a small-scale (32*32*3) deep learning module isadopted to scan and detect the window of a crack detected by eachlarge-scale deep learning module.

As an improvement of the present invention, for detection by alarge-scale deep learning model, because the input of the model is a224*224*3 image and the image size will not exactly be an integralmultiple of 224, scaling the image will result in loss of information inthe image. Therefore, during scanning and detection, windows for eachlayer start from four right angles and slide to a center by 112 pixelseach time in x and y directions, wherein the center refers to theintersection of x axis and y axis that lead to an image symmetry, andscanning windows for one layer are encrypted to ensure that the wholeimage can be scanned in a partially overlapping manner and a rangecontaining all cracks is output; and for each detected 224*224 windowcontaining a crack, a small-scale deep learning model (32*32*3) isadopted to scan and detect each 224*224 image in a manner of sliding 16pixels each time in x and y directions respectively.

As an improvement of the present invention, on the basis of a narrowedrange of crack detection based on multi-scale deep learning, directthreshold segmentation will still lead to loss of micro-crackinformation. On the basis of a crack detection range based onmulti-scale deep learning, a method that combines median-filtered imagesubtraction, 1-scale Hessian matrix-based linear enhancement and Otsuthreshold segmentation is employed to carry out the preliminary crackextraction. The extracted information can contain full information ofboth macro-cracks and micro-cracks in an image.

The preliminary crack extraction method of “global-localco-segmentation” in step 2) is specifically as follows:

21) Within each 224*224 window where a crack is detected, the originalimage is converted into a gray-scale image and then processed bylarge-scale median filtering, the filtered image is subtracted from theoriginal image to obtain a new image, and Otsu adaptive thresholdsegmentation is applied to the new image to obtain a binary image img1of the crack.

22) The original image is processed by small-scale median filtering, andthe filtered image is subtracted from the original image to obtain a newimage, and then the linear portion of the new image is enhanced by aHessian matrix-based image enhancement method, wherein the Hessianmatrix at each point of the image

${{{I(x)}\mspace{14mu}{is}\mspace{14mu}{\nabla^{2}{I(x)}}} = \begin{bmatrix}{I_{xx}(x)} & {I_{xy}(x)} \\{I_{yx}(x)} & {I_{yy}(x)}\end{bmatrix}},$eigenvalues of the Hessian matrix at each point is calculated as λ₁ and

$\lambda_{2},{\lambda_{12} = \left\{ {{\begin{matrix}{{{\lambda_{2}}\left( {1 + \frac{\lambda_{1}}{\lambda_{2}}} \right)} = {{\lambda_{2}} + \lambda_{1}}} & {{{if}\mspace{14mu}\lambda_{2}} \leq \lambda_{1} \leq 0} \\{{{\lambda_{2}}\left( {1 - {\alpha\frac{\lambda_{1}}{\lambda_{2}}}} \right)} = {{\lambda_{2}} - {\alpha\lambda}_{1}}} & {{{{if}\mspace{14mu}\lambda_{2}} < 0 < \lambda_{1} < \frac{\lambda_{2}}{\alpha}},} \\0 & {otherwise}\end{matrix}{R(x)}} = {{\lambda_{12}(x)}\mspace{14mu}{is}\mspace{14mu}{defined}\mspace{14mu}{as}\mspace{14mu}{the}}} \right.}$enhanced image, and α=0.25. Otsu adaptive threshold segmentation isapplied to the enhanced image within each 32*32 window of a crackdetected by the small-scale deep learning model, and a binary image img2of the crack is obtained after uniting.

23) The binary image img1 and the binary image img2 are united to obtainan output crack binary image for the 224*224 window.

24) The output binary image results for all detected 224*224 windowscontaining cracks in the first step are united to obtain a preliminarycrack extraction result for the whole image.

As an improvement of the present invention, the fine crack detectionmethod based on deep learning, image segmentation and imagereconstruction in step 3) is specifically as follows:

As the preliminarily-extracted crack binary image will contain a lot ofnoise information, the proposed image segmentation method based on asingle-pixel crack skeleton is utilized to segment thepreliminarily-extracted crack image at two levels, and an imagereconstruction method, in connection with deep learning, is utilized toperform single-factor analysis on the segmented cracks to determinewhether they are cracks, ultimately realizing the fine extraction ofcracks.

The image segmentation method based on a single-pixel skeleton isconducted at two levels. The skeleton is first segmented according toconnectivity, portions of the crack binary image that each correspondsto one independent connected component are extracted respectively, andthen each remaining connected component is further segmented intoindependent portions with endpoint and node information collected fromthe single-pixel crack skeleton, and the first level of segmentation canbe implemented by utilizing the connectivity of the initial crack binaryimage.

The second level of segmentation is specifically as follows:

A pixel on the skeleton that has only one of its eight neighbors on theskeleton is defined as an endpoint, and a pixel that has more than threeof its eight neighbors on the skeleton is defined as a node, and on thebasis of each independent connected component skeleton, the binary imageoutput at the previous level is further segmented, with the endpoint andnode information of a single pixel, into independent portions as outputsof this level, and the method for segmenting an independent connectedcomponent includes the following operations:

a. An eight-neighbor rule is utilized to identify endpoints and nodes onthe skeleton, segmentation is terminated when there are only endpointsand no node on the skeleton, and the corresponding regions in theinitial crack binary image are directly output.

b. The nodes on the connected component are deleted, by utilizing theendpoint and node information, to obtain independent skeleton segments,the length of each skeleton segment is calculated, and a threshold T isset as 32.

c. As other interferences on cracks often occur in the form of skeletonbranches, for each segmented skeleton segment that contains one endpointfor the original skeleton, regardless of whether the length of theskeleton segment is lower than the threshold, a circle is drawn, with aradius of ten pixels and a center at the node, to obtain intersectionswith other skeleton branches of the same node, the region of theskeleton is determined with the angle bisector of each branch of thenode and the outline of the initial crack binary image, and independentcrack binary regions are obtained from segmentation and filling by aregion growing algorithm.

d. For a skeleton segment has a length greater than the threshold T andtwo ends as nodes on the original skeleton, a circle is drawn, with aradius of ten pixels and a center at the node, to obtain intersectionswith other skeleton branches of the same node, the region of theskeleton is determined with the angle bisector of the node and theoutline of the original binary image, and independent crack binaryregions are obtained from segmentation and filling by a region growingalgorithm.

e. If the length of the extracted skeleton is less than the threshold,the center of the skeleton is determined, and an initial crack binaryimage having a length and width as T around the center of the skeletonsegment is directly captured as a segmented image.

A single-factor analysis is performed, by the image reconstructionmethod in combination with deep learning, on the segmented cracks, and anew fine detection image that contains separately-segmented crackinformation but contains no other interference information needs to beconstructed for fine crack detection of each segmented independentregion.

As an improvement of the present invention, the specific method forconstructing a fine detection image in step 3) is as follows:

a. After an initial crack binary image is obtained, the regioncorresponding to the initial crack binary image is removed from theoriginal image.

b. The removed region of the original image is filled with the averagevalue of the three-channel (RGB) values of pixels around each connectedcomponent of the initial crack binary image, and the edge is removed bymedian filtering to construct a new image.

c. For images segmented from a crack binary image based on asingle-pixel skeleton at two levels, binary connected components areextracted, and the length of a bounding rectangle of each connectedcomponent is calculated.

d. A fine detection image is constructed with the original image, thenew image and the range of the bounding rectangles of connectedcomponents, that is, each independent segmented crack binary imageregion is filled with the value of the original image and a regionoutside the segmented crack binary image region is filled with the valueof the new image to obtain a constructed detection image, wherein thespecific rule is as follows: if the length of the bounding rectangle isless than or equal to 32, a detection image is constructed with theoriginal image and the new image within a range of 32*32 around thecenter, and is scaled to the size of 224*224; if the length of thebounding rectangle is greater than 32 but less than 96, a new image isconstructed within a range of the length of the bounding rectanglearound the center, and is scaled to the size of 224*224; if the lengthof the bounding rectangle is greater than 96 but less than 224, a224*224 image is directly constructed around the center with the newimage and the original image; and if the length of the boundingrectangle is greater than 224, a detection image is constructed within arange of 224 pixels expanding outwardly from the bounding rectanglearound the center.

After a constructed detection image for the segmented crack is obtained,detection based on deep learning is performed once again to carry outsingle-factor analysis on the segmented cracks, thus achieving thepurpose of fine extraction. The constructed detection images are alsodivided into two levels according to the two levels of the segmentationmethod based on the single-pixel skeleton. The specific method forscanning and detecting the constructed fine detection image with deeplearning is as follows:

a. For images constructed at two levels, if the image size is less thanor equal to 224*224*3, the image is enlarged to the size of 224*224*3,and then detected with a deep learning model to directly determinewhether a crack is contained therein.

b. For constructed detection images with a size larger than 224*224*3,corresponding crack skeletons are extracted, and grids equidistantlyspaced by 32 pixels in x and y directions are constructed, and a newimage is scanned and detected with a intersections of a skeleton and acheckerboard as a center, wherein some of the intersections should beremoved, so that the distance between every two intersections is largerthan 16 pixels. The detection results are analyzed statistically. Forconstructed images at the first level, if the proportion of detectionresults representing that cracks are contained is greater than 0.2, thenit is considered that cracks are contained. For constructed images atthe second level, if the proportion of detection results representingthat cracks are contained is greater than 0.8, then it is consideredthat cracks are contained.

After three steps including crack range detection based on multi-scaledeep learning, preliminary crack binary image extraction and fine crackdetection, a crack binary image having high precision, high accuracy andhigh recall can be obtained.

The present invention provides a method for precisely measuring crackwidths based on a single-pixel crack skeleton and Zernike orthogonalmoments.

Conventional methods for calculating crack widths, which calculate thedistance between two pixels by “pixel counting”, works badly incalculating the width of a crack less than 5 pixels. Therefore, forcracks smaller than 5 pixels in images, the present invention proposes amethod for calculating crack widths based on a single-pixel skeleton andZernike orthogonal moments.

The method for calculating crack widths based on a single-pixel skeletonand Zernike orthogonal moments utilizes a single-pixel skeleton todetermine a rotation angle for an image. The image at this point isrotated, symmetrized and mirrored to obtain two images symmetrical withrespect to the y coordinate axis. The second-order orthogonal momentA′₂₀ and the fourth-order orthogonal moment A′₄₀ are calculatedrespectively for the two symmetrical images by utilizing orthogonalmoment templates. The widths of the two symmetrical images arerespectively calculated by a formula

${2l} = {2{\sqrt{\frac{{5A_{40}^{\prime}} + {3A_{20}^{\prime}}}{8A_{20}^{\prime}}}.}}$An average value is taken as crack width (unit: pixel) at this pointafter two error corrections are performed, and a final width of thecrack at this point is obtained according to a pixel resolution(mm/pixel). It is worth noting that the method proposed by the presentinvention actually calculates an average width of a crack within anorthogonal moment template range at a certain point on an image.

1) A single-pixel crack skeleton around a width calculation point isextracted, a rotation angle is determined for the image, an image withina range of 13*13 pixels around the width calculation point is captured,and preprocessing 1 is performed to obtain a gray-scale image (0-1) ofthe image, a matrix, with all elements of 13*13 being 1, is subtractedfrom the gray-scale image, so that the gray value of a crack pixel inthe image is high and the gray value of a background pixel is low, andthe maximum gray value MAX is recorded.

2) The image is rotated according to the skeleton direction, mirroringoperations are respectively performed, with the column where theaccumulated pixel value of cracks is the highest after rotation (i.e.,the brightest column) as a central column, to obtain two symmetricalimages, and smooth filtering is vertically applied to the two images.

3) It is checked whether the images obtained from mirroring meetrequirements for calculation (assuming that the line width is 5 and thebrightest is column 2, if one of the images does not meet conditions forcalculation after mirroring, further processing is required). Theaverage gray value within the range of a central column template is setas L, and the average gray value of each column on either side of thecentral column is set as L1, L2, L3 and L4 in sequence, wherein threeconditions are set as: |L4−L3|/|L2−L3|>=0.3, |L1−L4|/|L−L4|>=0.8, and|L4−L3|/|L−L4|>=0.1. If all the three conditions are met, the centralcolumn and one column on the left side of the central column aredeleted, and Mark=2 is recorded, otherwise, Mark=0 is recorded, andpreprocessing 2 is performed: a minimum gray value MIN within a range of7*7 around the image center is found, and the MIN is subtracted from theimage gray to construct a new image.

4) The existing orthogonal moment templates M20 and M40 are used insteadof the formula to calculate corresponding orthogonal moments and thuscalculate the width of a crack in the image; the first error correctionis as follows: principle error correction values are calculated byinterpolation according to the value of m=L1/L with reference to FIG. 8; the second error correction is as follows: the average gray valuewithin the range of the central column template of the image is dividedby the maximum gray value MAX of the original image subtracted by therecorded minimum gray value MIN to obtain a correction factor, and aMark value is added to the corrected value to obtain a final crack widthof each mirror image; and then an average value of the calculated widthsof the two mirror images after the error corrections is taken as a finalcrack width.

A device for automatically drawing cracks of a concrete specimen andmeasuring widths thereof in a laboratory comprises an image calibrationmodule, an image acquisition module, an image processing system, and acamera tripod.

The device for automatically drawing structural cracks and preciselymeasuring widths thereof can rapidly and automatically draw structuralcracks on a surface and precisely measure crack widths.

The image calibration module is configured to correct the pose of acamera, calculate the resolution of an image, and paste a plurality ofmanually-printed checkerboards with a known size on the surface of aconcrete specimen; the image acquisition module composed of a singlelens reflex camera is mainly configured to acquire a high-resolutionimage of the concrete specimen for image processing in the next step;and the image processing system is embedded with an automatic crackdrawing algorithm module, a crack feature analysis algorithm module, acalculation result storage module and a wireless transmission module,wherein the wireless transmission module is configured to wirelesslytransmit a detection result and an original image to a client.

The automatic crack drawing algorithm module comprises a module forcrack range detection based on multi-scale deep learning, a module forpreliminary crack extraction based on deep learning, median filteringand Hessian matrix-based linear enhancement, and a module for fine crackdetection based on deep learning, image segmentation and imagereconstruction, and can rapidly and automatically draw the acquiredimage with high precision, high accuracy and high recall, wherein theinformation of both macro-cracks and micro-cracks in the image is drawn.

The crack feature analysis algorithm module is configured to calculatecrack widths, and includes the method for calculating crack widths basedon a single-pixel skeleton and Zernike orthogonal moments proposed bythe present invention, and this method can also achieve highercalculation precision for micro-cracks smaller than 5 pixels in animage, obviously superior to conventional algorithms in terms ofcalculation precision.

Compared with the prior art, the present invention has the followingadvantages. 1) The device developed by the present invention can bewidely applied to the detection of structural cracks on a surface andthe detection of cracks occurred on a surface in a laboratory structuretest, and can output a precise crack binary image and preciselycalculate the length, width and other features of a crack in just a fewseconds; 2) The solution can rapidly, automatically and precisely draw astructural crack on a surface. The algorithm proposed by the presentinvention requires no human intervention to adjust the thresholdsegmentation coefficient and other coefficients, thereby resulting in awide application range and a higher automation level. In addition, bothmacro-cracks and micro-cracks in the image can be identified as many aspossible in a crack binary image output by the present invention, noiseinterferences, such as artificial marks and environmental pollution, canbe removed effectively and precisely, and the identified crack binaryimage has high precision, high accuracy and high recall. 3) The solutioncan precisely measure the width of a structural crack on a surface. Themethod for measuring crack widths proposed by the present inventiondemonstrates a measurement precision significantly higher than aconventional method for measuring crack widths, especially formicro-cracks smaller than 5 pixels in images, and the error of widthmeasurement for a micro-crack smaller than 5 pixels in an image can becontrolled to be less than 10%. Meanwhile, this means that under thesame measurement precision requirement, the method proposed by thepresent invention has a field of view nearly four times larger than thatof a conventional method.

To make the aforementioned more comprehensible, several embodimentsaccompanied with drawings are described in detail as follows.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the disclosure, and are incorporated in and constitutea part of this specification. The drawings illustrate exemplaryembodiments of the disclosure and, together with the description, serveto explain the principles of the disclosure.

FIG. 1 is a conceptual diagram of a method and device for automaticallydrawing structural cracks and precisely measuring widths thereofaccording to the present invention.

FIG. 2 is an overall technical roadmap according to the presentinvention.

FIG. 3 is a diagram for a method for automatically drawing cracksaccording to the present invention.

FIG. 4 is a diagram for a preliminary crack extraction method of“global-local co-segmentation” according to the present invention.

FIG. 5 is a diagram of a method for constructing a fine detection imagein fine crack detection according to the present invention.

FIG. 6 is a diagram of a method for calculating widths of micro-cracksin an image based on a single-pixel skeleton and Zernike orthogonalmoments according to the present invention.

FIG. 7 shows convolutional templates for calculating Zernike orthogonalmoments in micro-crack width measurement according to the presentinvention.

FIG. 8 is a graph of the first error correction (calculating principleerrors by interpolation) in crack width measurement according to thepresent invention.

FIG. 9 is an image of cracks on a concrete specimen shot by a camera inthe example according to the present invention.

FIG. 10 is a result image of crack ranges scanned and detected byutilizing deep learning (224*224*3) according to the present invention.

FIG. 11 is a result image of preliminary crack binary image extractionaccording to the present invention.

FIG. 12 is a result image of fine crack binary image extractionaccording to the present invention.

FIG. 13 is a result image of manually drawing a crack binary imageaccording to the present invention.

FIG. 14 is a result image of detection after directly applying Otsuadaptive threshold segmentation according to the present invention.

FIG. 15 is a result image of Otsu adaptive threshold segmentationdetection after Hessian matrix-based linear enhancement according to thepresent invention.

FIG. 16 is a result image of crack width measurement according to thepresent invention.

FIG. 17 is a graph of statistical comparative analysis for micro-crackwidth measurement results of different methods according to the presentinvention.

DESCRIPTION OF THE EMBODIMENTS

In order to deepen the understanding of the present invention, thepresent examples will be described in detail below with reference to theaccompanying drawings.

Example 1

A method for automatically drawing structural cracks As shown in FIG. 3, the method for automatically drawing structural cracks comprises thefollowing steps: 1) detecting, by a crack detection method based onmulti-scale deep learning, a crack range in an image using multi-scaledeep learning to ensure that no crack information is missed forsubsequent work; 2) extracting, by a preliminary crack extraction methodof “global-local co-segmentation” based on deep learning, medianfiltering and Hessian matrix-based linear enhancement, full informationof both macro-cracks and micro-cracks in the image to ensure a detectionresult of high recall; and 3) effectively removing, by a fine crackdetection method based on deep learning, image segmentation and imagereconstruction, noise from the result to ensure a detection result ofhigh accuracy and precision. The crack detection method based onmulti-scale deep learning in step 1) is specifically as follows: Becauseconcrete cracks are complex and diversified in morphology, single-scaledeep learning often cannot meet requirements for detecting cracks ofdifferent widths in an image, and crack information in an image can beeasily missed in the analysis based on detection results of single-scaledeep learning. With deep learning models of multiple scales from largeto small, the large-scale deep learning modules are first adopted toscan and detect the whole image; and on the basis that a crack range hasbeen determined by the large-scale (224*224*3) deep learning modules, asmall-scale (32*32*3) deep learning module is adopted to scan and detectthe window of a crack detected by each large-scale deep learning module.For detection by a large-scale deep learning model, because the input ofthe model is a 224*224*3 image and the image size will not exactly be anintegral multiple of 224, scaling the image will result in loss ofinformation in the image. Therefore, during scanning and detection,windows for each layer start from four right angles and slide to acenter by 112 pixels each time in x and y directions, wherein the centerrefers to the intersection of x axis and y axis that lead to an imagesymmetry, and scanning windows for one layer are encrypted to ensurethat the whole image can be scanned in a partially overlapping mannerand a range containing all cracks is output; and for each detected224*224 window containing a crack, a small-scale deep learning model(32*32*3) is adopted to scan and detect each 224*224 image in a mannerof sliding 16 pixels each time in x and y directions respectively. Onthe basis of a narrowed range of crack detection based on multi-scaledeep learning, direct threshold segmentation will still lead to loss ofmicro-crack information. On the basis of a crack detection range basedon multi-scale deep learning, a method that combines median-filteredimage subtraction, 1-scale Hessian matrix-based linear enhancement andOtsu threshold segmentation is employed to carry out the preliminarycrack extraction. The extracted information can contain full informationof both macro-cracks and micro-cracks in an image.

The preliminary crack extraction method of “global-localco-segmentation” in step 2) is shown in FIG. 4 and is specifically asfollows:

21) Within each 224*224 window where a crack is detected, the originalimage is converted into a gray-scale image and then processed bylarge-scale median filtering, the filtered image is subtracted from theoriginal image to obtain a new image, and Otsu adaptive thresholdsegmentation is applied to the new image to obtain a binary image img1of the crack.

22) The original image is processed by small-scale median filtering, andthe filtered image is subtracted from the original image to obtain a newimage, and then the linear portion of the new image is enhanced by aHessian matrix-based image enhancement method, wherein the Hessianmatrix at each point of the image I(x) is

${{\nabla^{2}{I(x)}} = \begin{bmatrix}{I_{xx}(x)} & {I_{xy}(x)} \\{I_{yx}(x)} & {I_{yy}(x)}\end{bmatrix}},$eigenvalues of the Hessian matrix at each point is calculated as λ₁ andλ₂,

$\lambda_{12} = \left\{ {{\begin{matrix}{{{\lambda_{2}}\left( {1 + \frac{\lambda_{1}}{\lambda_{2}}} \right)} = {{\lambda_{2}} + \lambda_{1}}} & {{{if}\mspace{14mu}\lambda_{2}} \leq \lambda_{1} \leq 0} \\{{{\lambda_{2}}\left( {1 - {\alpha\frac{\lambda_{1}}{\lambda_{2}}}} \right)} = {{\lambda_{2}} - {\alpha\lambda}_{1}}} & {{{{if}\mspace{14mu}\lambda_{2}} < 0 < \lambda_{1} < \frac{\lambda_{2}}{\alpha}},} \\0 & {otherwise}\end{matrix}{R(x)}} = {\lambda_{12}(x)}} \right.$is defined as the enhanced image, and α=0.25. Otsu adaptive thresholdsegmentation is applied to the enhanced image within each 32*32 windowof a crack detected by the small-scale deep learning model, and a binaryimage img2 of the crack is obtained after uniting.

23) The binary image img1 and the binary image img2 are united to obtainan output crack binary image for the 224*224 window.

24) The output binary image results for all detected 224*224 windowscontaining cracks in the first step are united to obtain a preliminarycrack extraction result for the whole image.

As an improvement of the present invention, the fine crack detectionmethod based on deep learning, image segmentation and imagereconstruction in step 3) is specifically as follows:

As the preliminarily-extracted crack binary image will contain a lot ofnoise information, the proposed image segmentation method based on asingle-pixel crack skeleton is utilized to segment thepreliminarily-extracted crack image at two levels, and an imagereconstruction method, in connection with deep learning, is utilized toperform single-factor analysis on the segmented cracks to determinewhether they are cracks, ultimately realizing the fine extraction ofcracks.

The image segmentation method based on a single-pixel skeleton isconducted at two levels. The skeleton is first segmented according toconnectivity, portions of the crack binary image that each correspondsto one independent connected component are extracted respectively, andthen each remaining connected component is further segmented intoindependent portions with endpoint and node information collected fromthe single-pixel crack skeleton, and the first level of segmentation canbe implemented by utilizing the connectivity of the initial crack binaryimage.

The second level of segmentation is specifically as follows:

A pixel on the skeleton that has only one of its eight neighbors on theskeleton is defined as an endpoint, and a pixel that has more than threeof its eight neighbors on the skeleton is defined as a node, and on thebasis of each independent connected component skeleton, the binary imageoutput at the previous level is further segmented, with the endpoint andnode information of a single pixel, into independent portions as outputsof this level, and the method for segmenting an independent connectedcomponent includes the following operations:

a. An eight-neighbor rule is utilized to identify endpoints and nodes onthe skeleton, segmentation is terminated when there are only endpointsand no node on the skeleton, and the corresponding regions in theinitial crack binary image are directly output.

b. The nodes on the connected component are deleted, by utilizing theendpoint and node information, to obtain independent skeleton segments,the length of each skeleton segment is calculated, and a threshold T isset as 32.

c. As other interferences on cracks often occur in the form of skeletonbranches, for each segmented skeleton segment that contains one endpointfor the original skeleton, regardless of whether the length of theskeleton segment is lower than the threshold, a circle is drawn, with aradius of ten pixels and a center at the node, to obtain intersectionswith other skeleton branches of the same node, the region of theskeleton is determined with the angle bisector of each branch of thenode and the outline of the initial crack binary image, and independentcrack binary regions are obtained from segmentation and filling by aregion growing algorithm.

d. For a skeleton segment has a length greater than the threshold T andtwo ends as nodes on the original skeleton, a circle is drawn, with aradius of ten pixels and a center at the node, to obtain intersectionswith other skeleton branches of the same node, the region of theskeleton is determined with the angle bisector of the node and theoutline of the original binary image, and independent crack binaryregions are obtained from segmentation and filling by a region growingalgorithm.

e. If the length of the extracted skeleton is less than the threshold,the center of the skeleton is determined, and an initial crack binaryimage having a length and width as T around the center of the skeletonsegment is directly captured as a segmented image.

A single-factor analysis is performed, by the image reconstructionmethod in combination with deep learning, on the segmented cracks, and anew fine detection image that contains separately-segmented crackinformation but contains no other interference information needs to beconstructed for fine crack detection of each segmented independentregion.

The specific method for constructing a fine detection image in step 3)is as follows:

a. After an initial crack binary image is obtained, the regioncorresponding to the initial crack binary image is removed from theoriginal image.

b. The removed region of the original image is filled with the averagevalue of the three-channel (RGB) values of pixels around each connectedcomponent of the initial crack binary image, and the edge is removed bymedian filtering to construct a new image.

c. For images segmented from a crack binary image based on asingle-pixel skeleton at two levels, binary connected components areextracted, and the length of a bounding rectangle of each connectedcomponent is calculated.

d. A fine detection image is constructed with the original image, thenew image and the range of the bounding rectangles of connectedcomponents, that is, each independent segmented crack binary imageregion is filled with the value of the original image and a regionoutside the segmented crack binary image region is filled with the valueof the new image to obtain a constructed detection image, wherein thespecific rule is as follows: if the length of the bounding rectangle isless than or equal to 32, a detection image is constructed with theoriginal image and the new image within a range of 32*32 around thecenter, and is scaled to the size of 224*224; if the length of thebounding rectangle is greater than 32 but less than 96, a new image isconstructed within a range of the length of the bounding rectanglearound the center, and is scaled to the size of 224*224; if the lengthof the bounding rectangle is greater than 96 but less than 224, a224*224 image is directly constructed around the center with the newimage and the original image; and if the length of the boundingrectangle is greater than 224, a detection image is constructed within arange of 224 pixels expanding outwardly from the bounding rectanglearound the center.

After a constructed detection image for the segmented crack is obtained,detection based on deep learning is performed once again to carry outsingle-factor analysis on the segmented cracks, thus achieving thepurpose of fine extraction. The constructed detection images are alsodivided into two levels according to the two levels of the segmentationmethod based on the single-pixel skeleton, and the method forconstructing a fine detection image is shown in FIG. 5 . The specificmethod for scanning and detecting the constructed fine detection imagewith deep learning is as follows:

a. For images constructed at two levels, if the image size is less thanor equal to 224*224*3, the image is enlarged to the size of 224*224*3,and then detected with a deep learning model to directly determinewhether a crack is contained therein.

b. For constructed detection images with a size larger than 224*224*3,corresponding crack skeletons are extracted, and grids equidistantlyspaced by 32 pixels in x and y directions are constructed, and a newimage is scanned and detected with a intersections of a skeleton and acheckerboard as a center, wherein some of the intersections should beremoved, so that the distance between every two intersections is largerthan 16 pixels. The detection results are analyzed statistically. Forconstructed images at the first level, if the proportion of detectionresults representing that cracks are contained is greater than 0.2, thenit is considered that cracks are contained. For constructed images atthe second level, if the proportion of detection results representingthat cracks are contained is greater than 0.8, then it is consideredthat cracks are contained.

After three steps including crack range detection based on multi-scaledeep learning, preliminary crack binary image extraction and fine crackdetection, a crack binary image having high precision, high accuracy andhigh recall can be obtained.

Example 2

A method for precisely measuring crack widths based on a single-pixelcrack skeleton and Zernike orthogonal moments Conventional methods forcalculating crack widths, which calculate the distance between twopixels by “pixel counting”, works badly in calculating the width of acrack less than 5 pixels. Therefore, for cracks smaller than 5 pixels inimages, the present invention proposes a method for calculating crackwidths based on a single-pixel skeleton and Zernike orthogonal moments,as shown in FIG. 6 .

The method for calculating crack widths based on a single-pixel skeletonand Zernike orthogonal moments utilizes a single-pixel skeleton todetermine a rotation angle for an image. The image at this point isrotated, symmetrized and mirrored to obtain two images symmetrical withrespect to the y coordinate axis. The second-order orthogonal momentA′₂₀ and the fourth-order orthogonal moment A′₄₀ are calculatedrespectively for the two symmetrical images by utilizing Orthogonalmoment templates. The widths of the two symmetrical images arerespectively calculated by a formula

${2l} = {2{\sqrt{\frac{{5A_{40}^{\prime}} + {3A_{20}^{\prime}}}{8A_{20}^{\prime}}}.}}$An average value is taken as crack width (unit: pixel) at this pointafter two error corrections are performed, and a final width of thecrack at this point is obtained according to a pixel resolution(mm/pixel). It is worth noting that the method proposed by the presentinvention actually calculates an average width of a crack within anorthogonal moment template range at a certain point on an image.

1) A single-pixel crack skeleton around a width calculation point isextracted, a rotation angle is determined for the image, an image withina range of 13*13 pixels around the width calculation point is captured,and preprocessing 1 is performed to obtain a gray-scale image (0-1) ofthe image, a matrix, with all elements of 13*13 being 1, is subtractedfrom the gray-scale image, so that the gray value of a crack pixel inthe image is high and the gray value of a background pixel is low, andthe maximum gray value MAX is recorded.

2) The image is rotated according to the skeleton direction, mirroringoperations are respectively performed, with the column where theaccumulated gray value of cracks is the highest after rotation (i.e.,the brightest column) as a central column, to obtain two symmetricalimages, and smooth filtering is vertically applied to the two images.

3) It is checked whether the images obtained from mirroring meetrequirements for calculation (assuming that the line width is 5 and thebrightest is column 2, if one of the images does not meet conditions forcalculation after mirroring, further processing is required). Theaverage gray value within the range of a central column template is setas L, and the average gray value of each column on either side of thecentral column is set as L1, L2, L3 and L4 in sequence, wherein threeconditions are set as: |L4−L3|/|L2−L3|>=0.3, |L1−L4|/|L−L4|>=0.8, and|L4−L3|/|L−L4|>=0.1. If all the three conditions are met, the centralcolumn and one column on the left side of the central column aredeleted, and Mark=2 is recorded, otherwise, Mark=0 is recorded, andpreprocessing 2 is performed: a minimum gray value MIN within a range of7*7 around the image center is found, and the MIN is subtracted from theimage gray to construct a new image.

4) The existing orthogonal moment templates M20 and M40 as shown in FIG.7 are used instead of the formula to calculate corresponding orthogonalmoments and thus calculate the width of a crack in the image; the firsterror correction is as follows: principle error correction values arecalculated by interpolation according to the value of m=L1/L withreference to FIG. 8 ; the second error correction is as follows: theaverage gray value within the range of the central column template ofthe image is divided by the maximum gray value MAX of the original imagesubtracted by the recorded minimum gray value MIN to obtain a correctionfactor, and a Mark value is added to the corrected value to obtain afinal crack width of each mirror image; and then an average value of thecalculated widths of the two mirror images after the error correctionsis taken as a final crack width.

Example 3

A device for automatically drawing cracks of a concrete specimen andmeasuring widths thereof in a laboratory As shown in FIG. 1 , the devicecomprises an image calibration module 1, an image acquisition module 2,an image processing system 3, and a camera tripod 4.

The device for automatically drawing structural cracks and preciselymeasuring widths thereof can rapidly and automatically draw structuralcracks on a surface and precisely measure crack widths.

The image calibration module 1 is configured to correct the pose of acamera, calculate the resolution of an image, and paste a plurality ofmanually-printed checkerboards with a known size on the surface of aconcrete specimen; the image acquisition module 2 composed of a singlelens reflex camera is mainly configured to acquire a high-resolutionimage of the concrete specimen for image processing in the next step;and the image processing system 3 is embedded with an automatic crackdrawing algorithm module, a crack feature analysis algorithm module, acalculation result storage module and a wireless transmission module,wherein the wireless transmission module is configured to wirelesslytransmit a detection result and an original image to a client.

The automatic crack drawing algorithm module comprises a module forcrack range detection based on multi-scale deep learning, a module forpreliminary crack extraction based on deep learning, median filteringand Hessian matrix-based linear enhancement, and a module for fine crackdetection based on deep learning, image segmentation and imagereconstruction, and can rapidly and automatically draw the acquiredimage with high precision, high accuracy and high recall, wherein theinformation of both macro-cracks and micro-cracks in the image is drawn.

The crack feature analysis algorithm module is configured to calculatecrack widths, and includes the method for calculating crack widths basedon a single-pixel skeleton and Zernike orthogonal moments proposed bythe present invention, and this method can also achieve highercalculation precision for micro-cracks smaller than 5 pixels in animage, obviously superior to conventional algorithms in terms ofcalculation precision.

Application Example 1

The large-scale deep learning model with an input of 224*224*3 pixelsused in the present invention adopted the disclosed GoogLeNet model.There were 1000 types of output layers in the original network, whilethere were two types of output layers in a modified network, i.e., acrack type and a non-crack type. A transfer learning strategy and aself-constructed 224*224*3 image crack database were utilized to trainthe model. The deep learning model with an input of 32*32*3 pixels usedin the present invention adopted the disclosed ResNet 20 model, and likethe modified network, the original network had two types of outputlayers, i.e., the crack type and the non-crack type. A transfer learningstrategy and a self-constructed 32*32*3 image crack database wereutilized to train the model.

Before testing, manually printed checkerboards were pasted on thesurfaces or the same plane of a tested concrete specimen according tothe area thereof. The checkerboard size selected in this example was 4cm*4 cm, and the side length of each checker was 1 cm. The pastedcheckerboards were utilized to calibrate the pose of a camera, and thepose of the camera was adjusted so that each point on a checkerboardextracted from an image by a Harris corner detection method could bemaintained as a square on the image.

After the installation of the device was completed, a complete dataprocessing flow for each acquired image should comprise: scanning anddetecting with deep learning models to roughly determine crack ranges inthe image; preliminarily extracting a crack binary image; finelyextracting a crack binary image; outputting width information of eachmanually-selected measurement point; and saving the original image andthe output crack information.

Automatic Drawing of Crack Binary Image

Step 1, Crack range determination based on multi-scale deep learning Thetrained large-scale deep learning model with an input of 224*224*3pixels was utilized to scan and detect by sliding windows on theacquired image, to check whether there were cracks in the image, and toroughly determine crack ranges in the image, and then marking wasperformed; and the trained small-scale deep learning model with an inputof 32*32*3 pixels was utilized to scan each window of 224*224 pixelsmarked in the previous step in a moving manner, and marking wasperformed.

Step 2, Preliminary crack extraction The three-channel (RGB) originalimage was converted into a gray-scale image, each crack-contained224*224 image marked in step 1 was processed by large-scale medianfiltering and then was subtracted from the original image to obtain anew image. Otsu adaptive threshold segmentation was applied to the newimage to obtain a crack binary image img1. Within the range of eachimage of 224*224 pixels marked in step 2, the image was processed bysmall-scale median filtering and then was subtracted from the originalgray-scale image to obtain a new gray-scale image, then 1-scaleHessian-based linear enhancement was applied to the new gray-scale imageto obtain an enhanced image. Within the range of each 32*32 windowmarked in step 1, Otsu adaptive threshold segmentation was applied tothe enhanced image, and then uniting operation was performed to obtain acrack binary image img2. All the crack binary images img1 and the crackbinary images img2 were united to obtain an initial crack binary image.

Step 3, Fine crack extraction

Image thinning was applied to the initial crack binary image obtained instep 2 to obtain a crack skeleton with a single-pixel width, and thespecific steps were as follows:

a. Zhang's fast parallel thinning algorithm was first employed to thinthe initial crack binary image.

b. The texture thinned in the previous step was not a single pixel,because some of the points that should be deleted did not meet thedeletion condition of Zhang's thinning algorithm. Eight neighbors ofeach of these points were coded in a binary manner, and the codedneighbors were converted into decimal forms to obtain numbers 65, 5, 20,80, 13, 97, 22, 208, 67, 88, 52, 133, 141 and 54.

c. The eight neighbors of a target point were coded in a binary mannersequentially around a target point, and if the coding conforms to theaforementioned deletion feature, then the target point will be deletedto ensure the single-pixel property of an output target.

d. The eight-neighbor rule was utilized to identify endpoints (one pixelof its eight neighbors was 1) and nodes (three or four pixels of itseight neighbors were 1) on the skeleton, and the positions of the nodeswere saved.

e. The identified nodes were deleted, and the crack skeleton wassegmented into segments. Each segment contained branches of one or twoendpoints, the lengths of the skeleton branches were calculated, andthose branches that were smaller than 20 pixels were considered as burrsand removed. If a node had two skeleton branches and the lengths of thetwo branches, i.e., a distance from the node to a respective end point,both were less than 20 pixels, then the longer one remained.

f. With the identified nodes, the segmented skeletons were connected andthe isolated points were contained in the skeleton to obtain a crackskeleton of a single-pixel width, and then the endpoint and nodeinformation of the skeleton was re-identified.

After the single-pixel crack skeleton was obtained, the skeleton wasfirst segmented according to connectivity, portions of the crack binaryimage that each corresponds to one independent connected component wereextracted respectively. A detection image was constructed for eachportion in combination with the original image (RGB), and it wasdetected whether a crack was contained in the connected component onceagain by deep learning, and the connected components that did notcontain cracks were deleted.

Then, all the remaining connected components contained crackinformation, but some connected components might contain both cracks andnoise. With the endpoint and node information collected from thesingle-pixel crack skeleton, each remaining connected component wasfurther segmented into independent portions, and a detection image wasconstructed for each portion in combination with the original image(RGB). It was detected whether a segmented portion contained crackinformation once again by utilizing a deep learning model, and an outputcrack binary image was obtained after the fine detection was completed.

Measurement of Crack Width at One Point in Image

Step 1, The crack width at one point in the image was measured, and thenormal direction of the crack was determined with the single-pixel crackskeleton at this point. As the output crack binary image result would beaffected by the size of a median filtering template used in thepreliminary crack extraction, crack width L (unit: pixel) was roughlydetermined by utilizing the intersections of the normal direction andthe edge of the output final crack binary image. Within a rectangularrange with a side length of 5*L around the center of a measurementpoint, median filtering based on a template size of 2*L was applied, andthe filtered image was subtracted from the original image to obtain anew image. With Canny operator, the distance between the twointersections of the normal at this point and the edge of the new imagewas detected to determine crack width (unit: pixel) at this point. Thepixel resolution of the image was calculated from the number of pixelsoccupied by the checkerboard of a known size in the image, and the crackwidth (unit: pixel) was multiplied by the pixel resolution (mm/pixel) toobtain crack width (unit: mm). If the calculated width result wasgreater than 5 pixels, it was considered to meet the precisionrequirement.

Step 2, For a crack having a width less than 5 pixels calculated in step1, the width was calculated according to the crack width identificationmethod based on Zernike orthogonal moments proposed in the presentinvention.

Application Example 2

In an example where an experiment of destroying a concrete beam wasconducted, cracks on the concrete beam were automatically drawn andwidths of these cracks were measured according to the present invention.

(1) Experimental Preparation

Checkerboard marks were pasted on a concrete surface, and meanwhile, thecrack width measurement points were marked manually. In the presentexample, a plurality of checkerboards were pasted on the concretesurface, and the crack width measurement points were numbered manuallyto increase the identification difficulty for the algorithm, as shown inFIG. 9 . The camera was fixed by utilizing a tripod, and the pose of thecamera was corrected with the pasted checkerboards so that the cameracould directly shoot the front surface of the concrete beam, and a testsystem was connected to achieve the real-time calculation and real-timeanalysis.

(2) Scanning and Detection Based on Deep Learning

An image was scanned and detected by utilizing a trained large-scaledeep learning model with an input of 224*224*3. If the input image ofFIG. 9 was detected, the output result was shown in FIG. 10 .

(3) Extraction of Initial Crack Binary Image Based on Deep Learning

For each 224*224 window containing a crack detected in the previousstep, a 32*32*3 small-scale deep learning model was employed to scan anddetect each 224*224 image in a moving manner. The image was processed bylarge-scale median filtering, and the filtered image was subtracted fromthe original image to obtain a new image, and Otsu adaptive thresholdsegmentation was applied to the new image to obtain a crack binary imageimg1. Also, the image was processed by small-scale median filtering, andthe filtered image was subtracted from the original image to obtain anew image, the linear portion of the new image was then enhanced by aHessian matrix-based image enhancement method, Otsu adaptive thresholdsegmentation was applied to the enhanced image within each 32*32 windowof a crack detected by the small-scale model, and a crack binary imageimg2 was obtained after uniting. The binary image img1 and the binaryimage img2 are united to obtain an output crack binary image for the224*224 window. The above process was repeated, and the output binaryimage results for all detected 224*224 windows containing cracks in theprevious step were united to obtain a preliminary crack extractionresult for the whole image, as shown in FIG. 11 .

(4) Fine Detection of Crack Binary Image Based on Deep Learning

After the previous step, the single-pixel crack skeleton of the crackbinary image was first extracted, and on this basis, thepreliminarily-extracted crack binary image was segmented at two levels.The initial crack region was removed from the original image, and theremoved region was filled with an average value of the pixels aroundeach connected component to construct a new image. The crack binaryimage was segmented by utilizing the single-pixel crack skeleton, acorresponding fine detection image was constructed according to theoriginal image and the new image, it was detected whether there was acrack once again by utilizing a trained deep learning model, and thefine detection result was output, as shown in FIG. 12 .

The manually-drawn crack binary image as shown in FIG. 13 was comparedwith the crack binary image result output by the method provided in thepresent invention. The image was segmented into equidistantly spacedsquares, each of 25*25 pixels, and 38,400 squares in total.

The condition of the manually-drawn crack in each square was comparedwith that of the automatically-drawn crack to determine the detectionprecision.

TP (True Positive): The number of squares containing a crack in both themanually-drawn crack image and the automatically-drawn crack image;

TN (True Negative): The number of squares containing no crack in boththe manually-drawn crack image and the automatically-drawn crack image;

FP (False Positive): The number of squares containing no crack in themanually-drawn crack image but containing a crack in theautomatically-drawn crack image; and

FN (False Negative): The number of squares containing a crack in themanually-drawn crack image but containing no crack in theautomatically-drawn crack image;

wherein three indexes were set:accuracy=(TP+TN)/(TP+FN+FP+TN), precision=TP/(TP+FP), andrecall=TP/(TP+FN).

The extracted crack result was analyzed, and the corresponding accuracy,precision and recall were 99.82%, 93.45% and 92.9% respectively,indicating that the method provided in the present invention could stillmaintain high accuracy, high precision and high recall under theinterference of the complex artificial marks.

FIG. 14 is a detection result after directly applying Otsu adaptivethreshold segmentation.

FIG. 15 is a detection result after Otsu adaptive threshold segmentationfollowing Hessian matrix-based image enhancement applied to the originalimage under the scale of n=8.

As shown in FIG. 14 and FIG. 15 , it can be seen that the methodprovided by the present invention is significantly superior to aconventional method.

(5) Crack Width Measurement

The normal direction of the crack was determined by the single-pixelcrack skeleton at the crack width measurement point, and width L wasroughly determined by utilizing the intersections of the normaldirection and the edge of the output crack binary image. Within arectangular range with a side length of 5*L around the center of ameasurement point, median filtering based on a template size of 2*L wasapplied, and the filtered image was subtracted from the original imageto obtain a new image. With Canny operator, the two intersections of thenormal at this point and the edge of the new image were detected todetermine crack width (unit: pixel) at this point. If the width wasgreater than 5 pixels, it was considered to meet the precisionrequirement. The pixel resolution of the image was calculated byutilizing the checkerboard calibrator of a known size in the image, andthe crack width (unit: pixel) was multiplied by the pixel resolution(mm/pixel) to obtain crack width. For a crack having a calculated widthless than 5 pixels, the width at this point was calculated according tothe proposed crack width identification method based on a single-pixelskeleton and Zerniek orthogonal moments.

Crack width manually measured by using a crack width gauge with aprecision of 0.0125 mm was compared with that of the proposed method,and the comparison result was shown in FIG. 16 .

The advantages of the method provided by the present invention inmeasuring micro-crack widths in an image, over a conventionalmeasurement method by “calculating the distance between twointersections”, were further verified. A cracked concrete beam wasadopted as an experimental object, crack images were captured with acamera from the front at different distances, and 248 width calculationpoints were selected for cracks smaller than 5 pixels in differentimages. A pixel resolution at the position of a crack was calculatedaccording to a checkerboard near the crack. Meanwhile, images wereacquired by a crack width gauge, the crack on an image acquired by thecrack width gauge was drawn manually, and an average width of the crackin a corresponding length range was calculated as a true value.

Since the method provided in the present invention actually calculatedthe average width of the crack at a certain point on the image withinthe orthogonal moment template range (7*7 pixels), the average widthcalculated by the method provided in the present invention was comparedwith average width values calculated within orthogonal moment templateranges corresponding to crack binary images extracted under thefollowing twelve working conditions.

Working condition 1: An image was converted into a gray-scale image.After the gray-scale image was smoothed, Ostu threshold segmentation wasdirectly applied to the smoothed image to obtain a binary image, and theaverage width of a crack was calculated within the range of 7*7 pixelsalong the length direction of the crack at this point.

Working condition 2: An image was converted into a gray-scale image, andmedian filtering was applied to the gray-scale image. Then the filteredimage was subtracted from the original gray-scale image to obtain a newimage. Ostu threshold segmentation was applied to the new image toobtain a binary image, and the average width of a crack was calculatedwithin the range of 7*7 pixels along the length direction of the crackat this point.

Working conditions 3 to 6: An image was converted into a gray-scaleimage, and median filtering was applied to the gray-scale image. Thenthe filtered image was subtracted from the original gray-scale image toobtain a new image. linear enhancement was applied to the new image,with a scale of n=1, n=2, n=4 and n=10 for work conditions 3, 4, 5, 6respectively. Ostu threshold segmentation was applied to the enhancedimage to obtain a binary image, and the average width of a crack wascalculated within the range of 7*7 pixels along the length direction ofthe crack at this point.

Working condition 7: An image was converted into a gray-scale image. Theedge was detected by Canny operator, and image closing operation wasperformed to obtain a binary image. The average width of a crack wascalculated within the range of 7*7 pixels along the length direction ofthe crack at this point.

Working condition 8: An image was converted into a gray-scale image. Theedge was detected by Canny operator, image closing operation wasperformed to obtain a new image, and the edge extracted by Cannyoperator was then subtracted from the new image to obtain a binaryimage. The average width of a crack was calculated within the range of7*7 pixels along the length direction of the crack at this point.

Working condition 9: The results of working condition 7 and workingcondition 8 were averaged.

Working condition 10: An image was converted into a gray-scale image,and median filtering was applied to the gray-scale image. Then thefiltered image was subtracted from the original gray-scale image toobtain a new image. The edge of the new image was detected by Cannyoperator, and then image closing operation was performed to obtain abinary image. The average width of a crack was calculated within therange of 7*7 pixels along the length direction of the crack at thispoint.

Working condition 11: An image was converted into a gray-scale image,and median filtering was applied to the gray-scale image. Then thefiltered image was subtracted from the original gray-scale image toobtain a new image. The edge of the new image was detected by Cannyoperator. The image closing operation was performed, and the edgeextracted by Canny operator was then subtracted from the image resultingfrom image closing operation to obtain a binary image. The average widthof a crack was calculated within the range of 7*7 pixels along thelength direction of the crack at this point.

Working condition 12: The results of working condition 10 and workingcondition 11 were averaged.

Error statistics was performed on the results calculated by differentmethods, and the error statistics results were shown in Table 1.

TABLE 1 Comparative Statistical Analysis of Proposed Methods WorkingWorking Working Working Working Working Serial Proposed ConditionCondition Condition Condition Condition Condition Number Method 1 2 3 45 6 Expected 8.34% 22.95% 18.36% 16.50% 14.31% 59.39% 66.03% ErrorStandard 6.32% 19.36% 14.19% 8.56% 11.70% 28.54% 29.78% Deviation ofErrors Working Working Working Working Working Working Serial ConditionCondition Condition Condition Condition Condition Number 7 8 9 10 11 12Expected 55.78% 16.60% 21.87% 52.88% 16.60% 19.37% Error Standard 24.11%8.21% 16.89% 24.22% 12.25% 17.08% deviation of errors

Working condition 1 and working condition 2 were grouped together asthey adopted simple threshold segmentation. Working conditions 3 to 6were grouped together as they adopted linear enhancements of differentscales. Working conditions 8 to 12 were grouped together because thewidth was calculated after edge extraction based on Canny operator. Theprobability density distribution of the best performing workingcondition in each group was fitted correspondingly, and was comparedwith the result of the proposed method, and the result showed that themethod provided in the present invention could achieve higher precisionfor micro-crack width measurement in comparison with the conventionalwidth measurement methods. The comparison results were shown in FIG. 17.

To sum up, with the method and device for automatically drawingstructural cracks and precisely measuring widths thereof provided in thepresent invention, a qualitative and quantitative analysis had beenperformed on cracks of the surface of a concrete specimen, and theprocess comprised: (1) rapidly and automatically drawing a crack binaryimage with high precision; and (2) precisely calculating crack width.Moreover, the method achieved significantly higher precision thanconventional methods in measuring micro-cracks in an image.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed embodimentswithout departing from the scope or spirit of the disclosure. In view ofthe foregoing, it is intended that the disclosure covers modificationsand variations provided that they fall within the scope of the followingclaims and their equivalents.

What is claimed is:
 1. A method for automatically drawing structuralcracks, comprising the following steps: 1) Detecting, by a crackdetection method based on multi-scale deep learning, a crack range in animage using multi-scale deep learning; 2) Extracting, by a preliminarycrack extraction method of “global-local co-segmentation” based on deeplearning, median filtering and Hessian matrix-based linear enhancement,information of both macro-cracks and micro-cracks in the image; and 3)Effectively removing, by a fine crack detection method based on deeplearning, image segmentation and image reconstruction, noise from theresult.
 2. The method for automatically drawing structural cracksaccording to claim 1, wherein the crack detection method based onmulti-scale deep learning in step 1) comprises: with deep learningmodels of multiple scales from large to small, the large-scale deeplearning modules are first adopted to scan and detect the whole image;and on the basis that a crack range has been determined by thelarge-scale (224*224*3) deep learning modules, a small-scale (32*32*3)deep learning module is adopted to scan and detect the window of a crackdetected by each large-scale deep learning module, wherein: for scanningand detection by the large-scale deep learning models, windows for eachlayer start from four right angles and slide to a center by 112 pixelseach time in x and y directions, wherein the center refers to theintersection of x axis and y axis that lead to an image symmetry, andscanning windows for one layer are encrypted; and for each detected224*224 window containing a crack, a small-scale deep learning model(32*32*3) is adopted to scan and detect each 224*224 image in a mannerof sliding 16 pixels each time in x and y directions respectively. 3.The method for automatically drawing structural cracks according toclaim 2, wherein the preliminary crack extraction method of“global-local co-segmentation” in step 2) comprises: 21) within each224*224 window of a crack detected by the large-scale deep learningmodel, an original image is converted into a gray-scale image and thenprocessed by large-scale median filtering, the filtered image issubtracted from the original image to obtain a new image, and Otsuadaptive threshold segmentation is applied to the new image to obtain abinary image img1 of the crack; 22) the original image is processed bysmall-scale median filtering, and the filtered image is subtracted fromthe original image to obtain a new image, and then the linear portion ofthe new image is enhanced by a Hessian matrix-based image enhancementmethod, wherein the Hessian matrix at each point of the image I(x) is${{\nabla^{2}{I(x)}} = \begin{bmatrix}{I_{xx}(x)} & {I_{xy}(x)} \\{I_{yx}(x)} & {I_{yy}(x)}\end{bmatrix}},$ eigenvalues of the Hessian matrix at each point iscalculated as λ₁ and λ₂, $\lambda_{12} = \left\{ {{\begin{matrix}{{{\lambda_{2}}\left( {1 + \frac{\lambda_{1}}{\lambda_{2}}} \right)} = {{\lambda_{2}} + \lambda_{1}}} & {{{if}\mspace{14mu}\lambda_{2}} \leq \lambda_{1} \leq 0} \\{{{\lambda_{2}}\left( {1 - {\alpha\frac{\lambda_{1}}{\lambda_{2}}}} \right)} = {{\lambda_{2}} - {\alpha\lambda}_{1}}} & {{{{if}\mspace{14mu}\lambda_{2}} < 0 < \lambda_{1} < \frac{\lambda_{2}}{\alpha}},} \\0 & {otherwise}\end{matrix}{R(x)}} = {\lambda_{12}(x)}} \right.$ is defined as theenhanced image, and α=0.25, Otsu adaptive threshold segmentation isapplied to the enhanced image within each 32*32 window of a crackdetected by the small-scale deep learning model, and a binary image img2of the crack is obtained after uniting; 23) the binary image img1 andthe binary image img2 are united to obtain an output crack binary imagefor the 224*224 window; 24) the output binary image results for alldetected 224*224 windows containing cracks in the first step are unitedto obtain a preliminary crack extraction result for the whole image. 4.The method for automatically drawing structural cracks according toclaim 3, wherein the fine crack detection method based on deep learning,image segmentation and image reconstruction in step 3) comprises: aproposed image segmentation method based on a single-pixel crackskeleton is utilized to segment the preliminarily-extracted crack imageat two levels, and an image reconstruction method, in connection withdeep learning, is utilized to perform single-factor analysis onsegmented cracks to determine whether they are cracks, ultimatelyrealizing the fine extraction of cracks, wherein the image segmentationmethod based on a single-pixel skeleton is conducted at two levels: theskeleton is first segmented according to connectivity, portions of thecrack binary image that each corresponds to one independent connectedcomponent are extracted respectively, and then each remaining connectedcomponent is further segmented into independent portions with endpointand node information collected from the single-pixel crack skeleton, andthe first level of segmentation can be implemented by utilizing theconnectivity of the initial crack binary image; the second level ofsegmentation comprises: a pixel on the skeleton that has only one of itseight neighbors on the skeleton is defined as an endpoint, and a pixelthat has more than three of its eight neighbors on the skeleton isdefined as a node, and on the basis of each independent connectedcomponent skeleton, the binary image output at the previous level isfurther segmented, with the endpoint and node information of a singlepixel, into independent portions as outputs of this level, and themethod for segmenting an independent connected component comprises thefollowing operations: a. an eight-neighbor rule is utilized to identifyendpoints and nodes on the skeleton, segmentation is terminated whenthere are only endpoints and no node on the skeleton, and thecorresponding regions in the initial crack binary image are directlyoutput; b. the nodes on the connected component are deleted, byutilizing the endpoint and node information, to obtain independentskeleton segments, the length of each skeleton segment is calculated,and a threshold T is set as 32; c. for each segmented skeleton segmentthat contains one endpoint for the original skeleton, regardless ofwhether the length of the skeleton segment is lower than the threshold,a circle is drawn, with a radius of ten pixels and a center at the node,to obtain intersections with other skeleton branches of the same node,the region of the skeleton is determined according to the angle bisectorof each branch of the node and the outline of the initial crack binaryimage, and independent crack binary regions are obtained fromsegmentation and filling by a region growing algorithm; d. for askeleton segment has a length greater than the threshold T and two endsas nodes on the original skeleton, a circle is drawn, with a radius often pixels and a center at the node, to obtain intersections with otherskeleton branches of the same node, the region of the skeleton isdetermined according to the angle bisector of the node and the outlineof the original binary image, and independent crack binary regions areobtained from segmentation and filling by a region growing algorithm; e.if the length of the extracted skeleton is less than the threshold, thecenter of the skeleton is determined, and an initial crack binary imagehaving a length and width as T around the center of the skeleton segmentis directly captured as a segmented image; and a single-factor analysisis performed, by the image reconstruction method in combination withdeep learning, on the segmented cracks, and a new fine detection imagethat contains separately-segmented crack information but contains noother interference information needs to be constructed for fine crackdetection of each segmented independent region.
 5. The method forautomatically drawing structural cracks according to claim 4, whereinthe specific method for constructing a fine detection image in step 3)comprises: a. after an initial crack binary image is obtained, theregion corresponding to the initial crack binary image is removed fromthe original image; b. the removed region of the original image isfilled with the average value of the three-channel (RGB) values ofpixels around each connected component of the initial crack binaryimage, and the edge is smoothed by median filtering to construct a newimage; c. for images segmented from a crack binary image based on asingle-pixel skeleton at two levels, binary connected components areextracted, and the length of a bounding rectangle of each connectedcomponent is calculated; d. a fine detection image is constructed withthe original image, the new image and the range of the boundingrectangles of connected components, that is, each independent segmentedcrack binary image region is filled with the value of the original imageand a region outside the segmented crack binary image region is filledwith the value of the new image to obtain a constructed detection image,wherein the specific rule is as follows: if the length of the boundingrectangle is less than or equal to 32, a detection image is constructedwith the original image and the new image within a range of 32*32 aroundthe center, and is scaled to the size of 224*224; if the length of thebounding rectangle is greater than 32 but less than 96, a new image isconstructed within a range of the length of the bounding rectanglearound the center, and is scaled to the size of 224*224; if the lengthof the bounding rectangle is greater than 96 but less than 224, a224*224 image is directly constructed around the center with the newimage and the original image; and if the length of the boundingrectangle is greater than 224, a detection image is constructed within arange of 224 pixels expanding outwardly from the bounding rectanglearound the center.